Skip to content

return a JSON clob as text: it was parsed as the envelope - #45

Merged
jjsaunier merged 2 commits into
masterfrom
fix/json-clob-decode
Nov 5, 2026
Merged

jjsaunier merged 2 commits into
masterfrom
fix/json-clob-decode

Conversation

@AbdElRhmanArafa

Copy link
Copy Markdown
Contributor

fix issue when return large object

─❯ python test-7/clob_json_repro.py                                                                                                                                                                                                 ─╯
WARNING:root:Params "session_sticky_proxy" is ignored. Works only if session is enabled
Traceback (most recent call last):
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/test-7/clob_json_repro.py", line 14, in <module>
    res = client.scrape(ScrapeConfig(
        url="https://www.eddiebauer.com/products.json?limit=250&page=1",
    ...<4 lines>...
        raise_on_upstream_error=False,
    ))
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/scrapfly/client.py", line 766, in scrape
    raise e
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/scrapfly/client.py", line 755, in scrape
    scrape_api_response = self._handle_response(response=response, scrape_config=scrape_config)
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/scrapfly/client.py", line 852, in _handle_response
    api_response = self._handle_api_response(
        response=response,
        scrape_config=scrape_config,
        raise_on_upstream_error=scrape_config.raise_on_upstream_error
    )
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/scrapfly/client.py", line 1352, in _handle_api_response
    api_response:ScrapeApiResponse = ScrapeApiResponse(
                                     ~~~~~~~~~~~~~~~~~^
        response=response,
        ^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        large_object_handler=self._handle_scrape_large_objects
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/scrapfly/api_response.py", line 441, in __init__
    self.result = self.handle_api_result(api_result=api_result)
                  ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/scrapfly/api_response.py", line 545, in handle_api_result
    api_result['result']['content'], api_result['result']['format'] = self.large_object_handler(callback_url=api_result['result']['content'], format=content_format)
                                                                      ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abdelrhman.arafa660.com/Documents/scrapfly-lab/.venv/lib/python3.15/site-packages/scrapfly/client.py", line 1310, in _handle_scrape_large_objects
    return content.decode('utf-8'), 'text'
           ^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'decode'

@jjsaunier
jjsaunier merged commit f887830 into master Nov 5, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants