Skip to content

Implement client base and storage endpoints - #23

Merged
fnesveda merged 10 commits into
masterfrom
feature/base-and-storage
Jan 14, 2021
Merged

fnesveda merged 10 commits into
masterfrom
feature/base-and-storage

Conversation

@fnesveda

@fnesveda fnesveda commented Dec 9, 2020

Copy link
Copy Markdown
Member

I've implemented the base of the client and the storage endpoints. It's modeled after the javascript client so the usage is similar. There's still a lot of documentation missing, and I need to implement some integration tests for these, but it should be good as a start of the project.

@mnmkng mnmkng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I obviously did not read the whole thing, because, well, I can't read Python very well. Great job @fnesveda anyway! This is a start of something great for Apify (and the world).

Comment thread src/apify_client/_utils.py Resolved

@mhamas mhamas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general looks quite good, left a bunch of comments to address / discuss. Thanks for kicking this out!

Comment thread README.md Resolved
Comment thread format.sh Resolved
Comment thread setup.py Resolved
Comment thread src/apify_client/_errors.py Resolved
Comment thread src/apify_client/_http_client.py Resolved
Comment thread tests/unit/test_utils.py Resolved
@fnesveda fnesveda self-assigned this Jan 8, 2021
@fnesveda

Copy link
Copy Markdown
Member Author

Thanks for the thorough review, @mhamas! I've addressed your comments, I agreed with most things and updated the code accordingly, the rest we already talked about on Friday. Can you please take another look?

@fnesveda
fnesveda requested a review from mhamas January 12, 2021 11:49

@mhamas mhamas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx @fnesveda

Comment thread README.md Resolved
Comment thread README.md Resolved
Comment thread lint_and_test.sh Resolved
Comment thread src/apify_client/_types.py Resolved
content_type = 'application/json; charset=utf-8'

if 'application/json' in content_type and not _is_file_or_bytes(value) and not isinstance(value, str):
value = json.dumps(value, indent=2)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, at least add big TODO here. From my previous experience the difference was couple of X for large JSONs. Thx

@Equidem Equidem left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Most of the stuff I would object to has already been mentioned, so I only have a few nitpicks.

"""Initialize the DatasetClient."""
super().__init__(*args, resource_path='datasets', **kwargs)

def get(self) -> Optional[Dict]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(applies to all the other clients like this too)
Maybe we could override the [] operator and allow extracting these values directly using the DatasetClient instance? It would be a little more user friendly and wouldn't require much additional work. We could also easily do the same for the setter use-case of [].

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be wrong conceptually - when you call client.dataset('someDatasetId'), you're getting another client, not the dataset itself. Implementing [] would mix these two concepts together, and I think in the end it would be just more confusing to users.

Also, we'd have to download the whole resource again and again for every property we would be accessing. I think that would be weird as well, when you call a function like .get(), you kinda intuitively expect that it might take some time, but when you access something via the [] operator, you intuitively expect it to be already there and accessible quickly.

We can revisit this later, but for this first version I'd skip it and stick just to the functions.

@fnesveda
fnesveda merged commit eebde2a into master Jan 14, 2021
@fnesveda
fnesveda deleted the feature/base-and-storage branch January 14, 2021 13:26
@fnesveda fnesveda linked an issue Mar 5, 2021 that may be closed by this pull request
@fnesveda fnesveda added this to the 4th Sprint 01/04-01/15 milestone Mar 5, 2021
@fnesveda fnesveda added the validated Issues that are resolved and their solutions fulfill the acceptance criteria. label May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

validated Issues that are resolved and their solutions fulfill the acceptance criteria.

Projects

None yet

5 participants