High-scale data transfers performance in Hubway Connect ? Watch Our MVP Joao Melo explain it in the video:
The common question is whether Hubway Connect will perform well at the high scale of data transfer. Performance is a hot topic when it comes to data staging, so it’s imperative to ensure that highly efficient measures are in place to avoid impacts on the overall quality of service in a runtime environment. We have placed performance and security at the top of our priorities list. Here are a few mechanisms we implemented to guarantee good performance:
- We refined the execution of queries against the transactional database by creating our own high-performance optimization engine. SQL queries are assembled and run using highly efficient algorithms.
- The output data handling was made as efficient as possible, which is extremely helpful when handling high-volume data.
- We used ROP strategies to load test the product to its limits, with scenarios involving over 1 million records. Our tests indicate that Hubway Connect’s data APIs perform even better than those created with traditional low-code development.
Check out our article outlining the test details, comparing an API created in OutSystems with a Hubway Connect API. The amount of records retrieved from a Hubway Connect API is limited, preventing a consumer from retrieving more data than is acceptable. We limit the output to 1,000 records by default, but an IT admin can configure this differently. Additionally, this setup provides an API that avoids failing with timeouts due to extensive waiting times.
Mechanisms like date range filtering and pagination can easily leverage out-of-the-box query options provided by the OData protocol. With this approach, data analysts, when staging data, can make use of an incremental load approach, which is a best practice standard. With Hubway Connect, you can rest assured that your runtime applications won’t be affected while your data staging pipeline will run smoothly with good response time.