Conversation
This is a draft to test unittest implementations for the server. Implement unittests using `werkzeug.test.Client` to test the `/shells` endpoint, as example. Each endpoint and method is tested for success and possible failures. The object store is reset prior to every test case. Tests are repeated for `application/json` and `application/xml` Content-Types. Therfore test are written against an abstract `FromatClient` that covers the details of (de-)serialization behind a simple API for requesting and parsing. Therefore the base class defining the test cases (`_ShellsEndpointTest`) is disabled for testing. Two subclasses are derived from this class, one for each format, that define the correct `FormatClient` and execute the tests.
…umbnail endpoint.
…ation thumbnail endpoint." This reverts commit 496190f, which held the content of eclipse-basyx#618. This was added for testing purpose only. Merge `develop` into this branch, after the PR was closed to obtain the same result.
For now the `/submodel-elements` paths are excluded
Added test class `TestPagination` to `test_base.py` that ensures pagination by following `cursor` value correctly assembles all items. Additionally, all endpoints, that should support pagination are checked if they do so.
To separate testing of the pagination logic from working endpoints, the shared function for creatin paginated responses is now tested directly. The base tests on paginated endpoints remain.
In the first version of the tests for the Discovery API, when data needed to be added to the DiscoveryStore, this was done through the `POST /lookup/shells/<aasId>` endpoint. To decouple endpoint tests from each other, the data insertion is now done directly via the DiscoveryStore.
…riptor writes" This reverts commit 46161d4. Creating issue for this to solve in later PR
Currently, the `DictDescriptorStore` used in the Registry when disabling persistent storage, throws an exception on `commit()` calls. To avoid failing pipeline and because persistent storage is more realistic end-user behavior, the integration tests now run with persistent storage.
…minor comment changes
Found issues eclipse-basyx#630, eclipse-basyx#631, eclipse-basyx#632 during test creation. We comment the code out, for now, to not block the PR by these issues. The issues hold instructions to uncomment these sections.
| STORE_FILE = os.path.join(os.path.dirname(__file__), "discovery_store_persistence_test.json") | ||
|
|
||
| def tearDown(self) -> None: | ||
| for path in (self.STORE_FILE, f"{self.STORE_FILE}.tmp"): | ||
| if os.path.exists(path): | ||
| os.remove(path) |
There was a problem hiding this comment.
@paul-gerber-svg You create the path of the store file here manually and additionally use the tearDown() method to define a cleanup. However, we only need the file during the time of one test function, so there is no need to define it this complicated.
I prefer using the builtin tempfile library. It offers a TemporaryDirectory class, that you can use with a with block. It needs no arguments and creates a temp directory at the correct location of your system. Inside this with block you can work with files and the cleanup is done automatically, when the with block is exited.
You can take a look at how I use it here: sdk/test/adapter/test_load_directory.py
| TEST_DIR = os.path.join(os.path.dirname(__file__), "load_directory_test_folder") | ||
|
|
||
| def setUp(self) -> None: | ||
| os.makedirs(self.TEST_DIR, exist_ok=True) | ||
| self.mock_endpoint = model.Endpoint( | ||
| interface="AAS-3.0", protocol_information=model.ProtocolInformation(href="https://example.org/") | ||
| ) | ||
|
|
||
| def tearDown(self) -> None: | ||
| shutil.rmtree(self.TEST_DIR) |
There was a problem hiding this comment.
@paul-gerber-svg The same as for the test_discovery.py holds here. You can use the tempfile.TemporaryDirectory() to make your life a lot easier.
| @staticmethod | ||
| def _descriptor_to_json_dict(desc: _Descriptor) -> Dict[str, Any]: | ||
| data: Dict[str, Any] = json.loads(json.dumps(desc, cls=adapter.ServerAASToJsonEncoder)) | ||
| data["modelType"] = DESCRIPTOR_TYPE_TO_STRING[type(desc)] | ||
| return data | ||
|
|
||
| def _write_file( | ||
| self, | ||
| filename: str, | ||
| aas_descriptors: Iterable[model.AssetAdministrationShellDescriptor] = (), | ||
| submodel_descriptors: Iterable[model.SubmodelDescriptor] = (), | ||
| ) -> None: | ||
| data = { | ||
| "assetAdministrationShellDescriptors": [self._descriptor_to_json_dict(d) for d in aas_descriptors], | ||
| "submodelDescriptors": [self._descriptor_to_json_dict(d) for d in submodel_descriptors], | ||
| } | ||
| with open(os.path.join(self.TEST_DIR, filename), "w") as f: | ||
| json.dump(data, f) |
There was a problem hiding this comment.
@paul-gerber-svg
I like that you created the _write_file() method to prevent duplication in the test arrangement. This is very helpful to keep the test cases itself clean and readable.
I would suggest here to get rid of the _descriptor_to_json_dict() method because the only usage is in the _write_file() method but it adds some complication with the DESCRIPTOR_TYPE_TO_STRING mapping and the _Descriptor type variable, that does not help here. You can just move all the logic of setting the data["modelType"] into _write_file() directly.
| aasd = model.AssetAdministrationShellDescriptor( | ||
| id_="https://example.org/AASDescriptor/1", endpoints=[self.mock_endpoint] | ||
| ) | ||
| sd = model.SubmodelDescriptor(id_="https://example.org/SubmodelDescriptor/1", endpoints=[self.mock_endpoint]) |
There was a problem hiding this comment.
@paul-gerber-svg I have create functions example_endpoint(), example_aas_descriptor() and example_submodel_descriptor() in server/test/adapter/descriptor_utils.py. It would be nice to use them here too. This way, you can also drop the self.mock_endpoint.
| self.assertIn("https://example.org/AASDescriptor/1", store) | ||
| self.assertIn("https://example.org/SubmodelDescriptor/1", store) | ||
|
|
||
| def test_merges_descriptors_from_multiple_files(self) -> None: |
There was a problem hiding this comment.
@paul-gerber-svg It would be nice for this and all following test cases to not only check for model.AssetAdministrationShellDescriptor but also model.SubmodelDescriptor. Can you add this to the test cases?
| self.assertEqual(1, len(store)) | ||
| self.assertIn("https://example.org/AASDescriptor/1", store) | ||
|
|
||
| def test_ignores_non_json_files(self) -> None: |
There was a problem hiding this comment.
@paul-gerber-svg Nice spot! I would have missed to test for this case ;)
3 participants
TODO
DiscoveryStoretest serde methodsfrom_file(), to_file()provider.py: functionload_directoryfor loadingDescriptorStorejsonization.py: Extendtest_registry.pycases to fullDescriptormodels (testing every branch of Encoder/Decoder)run_repository.py/run_registry.py: Testbuild_storage()respects environment variablesrun_discovery.py: Test loading file on run, saving on exitChanges
This PR adds unit and integration test to the previously untested server package. In detail the following test setup is used
Unit Testing
Unit tests are primarily written against the served API, employing
werkzeug.test.Clientto test thewerkzeugbased API interfaces.All three server interfaces (
repository,discovery,registry) are tested this way.Format specific testing
All implemented endpoints are tested with all supported
Content-Types and for all possible responses.Responses are not deserialized using the SDK adapter, because round-trip serialization-deserialization is already tested in the SDK. However, the correctness of responses are still verified by parsing them using standard libraries (
jsonandlxml.etree) and checking for specific expected data.To reduce repetition of test for both Content-Types
application/jsonandapplication/xml,test/interfaces/format_utils.pydefines a format-agnostic wrapperFormatClientofwerkzeug.test.Clientthat has subclassesJsonFormatClientandXmlFormatClienthandling serialization of request bodies as well as parsing and checking as described above.The test cases are written against this format-agnostic
FormatClient. Function decoratorswith_json_client/with_xml_clientare used to create separate test cases with the correct client at runtime.Coverage
In general, endpoints are tested for successful response and all error codes, defined in the specification. More specifically:
GETendpoints are tested forPOSTendpoints are tested for*ObjectStorePUTendpoints are tested for*ObjectStoreis altered correctlyDELETEendpoints are tested for*ObjectStoreAdditional Tests
While the test above test if the endpoints behavior is compliant to the specification, some additional tests directly test the code in
base.py:base.py, therefore we extensively test it separate for correct logic (pages build a partition of complete set) and parameter handling.JsonResponseandXmlResponseis tested separatelyHTTPApiDecoder), is already tested in the test cases above.Integration Testing
To ensure correct setup of uWSGI, nginx and supervisor inside the docker container, we run basic integration tests.
Therefore,
server/test/docker_integration/has one test module per profile (repository, registry, discovery) that runs against a real, already-running server, checking/descriptionand a full create/retrieve/delete roundtrip per profile.By default these tests skip if no server is reachable (like
test_couchdb.pyin the SDK). SettingREQUIRE_SERVER_INTEGRATION_TESTSmakes them fail instead of skip. We use this setting in theserver-dockerCI job, which now builds and tests all three profiles.