Why do we need a better way of integrating with Shopify?
In today's fast-paced digital landscape, businesses need to be able to integrate with e-commerce platforms like Shopify quickly and efficiently. For many businesses, however, developing integrations can be a time-consuming and expensive process.
For many businesses, building integrations in-house can be a significant investment in terms of time and money. It may require hiring additional developers or consultants, investing in new technology, and dedicating significant resources to the project. Additionally, building a custom integration can take many months, depending on the complexity of the integration and the availability of internal resources.
Given the complexity and cost of building integrations in-house, many businesses turn to digital agencies to help them develop integrations. By leveraging the agency's expertise and experience, they can have their integrations developed with less risk, however, it can still be very time-consuming.
The below diagram illustrates a custom middleware being built between your ERP system and Shopify. Note that the blue area reflects custom code that needs to be written for the integration.
Custom middleware/integration to Shopify is all work and no fun
As mentioned before, building a middleware or an integration like this has several disadvantages:
- High initial cost: Creating a middleware or integration from scratch can be a time-consuming and costly process. The entire integration project must be covered, from gathering requirements to design, development, testing, and deployment. The cost can include hiring specialized developers, purchasing hardware and software, and other infrastructure expenses. This upfront cost can be a significant barrier to entry for many companies, especially for smaller businesses with limited resources.
- High maintenance cost: Once the integration is built, ongoing maintenance costs can also be high. As systems change and evolve, the integration may need to be updated, modified, or even rebuilt entirely. Ongoing development, technical support, documentation, and training may be required to ensure that the integration continues to meet the business's needs. This can add to the overall cost and complexity of the project, making it difficult to manage.
- Requires a team of expert developers: Building a middleware or integration from scratch requires a team of expert developers with specialized skills and knowledge. Onboarding new team members can be difficult, as they need to understand the entire system and its underlying architecture. Additionally, developers with these skills and knowledge can be challenging to find, especially for smaller businesses that may not have the resources to compete with larger companies for top talent.
When a team of developers builds custom integrations for multiple clients, they often end up building similar integrations with unique nuances for each client. This leads to individual codebases for each integration, making future development and maintenance difficult and expensive.
Furthermore, having to maintain many custom integration projects can lead to the team being overwhelmed and unable to focus on delivering new business value to clients. This can lead to burnout, where the team becomes fatigued and demotivated, resulting in decreased productivity and lower-quality work.
A team trying to maintain many custom integrations simultaneously will have difficulty keeping up
The problem of burnout also affects the clients as the team may no longer be able to provide the same level of support and expertise they were known for, resulting in a decrease in the quality of service. Additionally, if key members of the team who knew everything about a particular client's project leave, the knowledge gap can create significant challenges for the client and may result in the need for additional training or hiring new staff to take over the project.
A better way of integrating into Shopify is needed because current integration methods such as building custom integrations can be costly, time-consuming, and difficult to maintain. These methods may also not be able to meet specific business requirements or adapt to changes in the e-commerce market. By using connectors specifically designed for Shopify, businesses can benefit from quick and easy integrations that meet their unique needs, while reducing costs and freeing up resources for other important tasks.
Can we build a composable middleware solution for Shopify?
Composable middleware solutions have become increasingly popular in recent years, as businesses seek to build more flexible and scalable integrations. These solutions use a modular approach to integration development, allowing businesses to connect multiple systems and services using a variety of pre-built components or "middleware."
For Shopify integrations, this could include middleware components for transferring orders to an ERP or OMS system, capturing or refunding order payments, managing shipping and fulfillment, and synchronizing products and customers.
If we could translate business needs for integrating with Shopify into generic functional requirements, then we could build a composable middleware solution for Shopify. The ideal result would allow us to maintain various integration flows towards Shopify independent of client nuances, while at the same time allowing us to communicate if any client system.
What does Shopify recommend?
Shopify recommends the following guidelines when building integrations to ensure a robust and scalable solution. One approach is to utilize webhooks to keep external systems up to date while separating webhook handling from data processing and transformation.
The Webhook Processor receives and stores webhooks from Shopify, verifying their digital signature to protect against forgery, and pushes them and their associated data to the Queue. The Queue maintains a list of webhook data for processing, while the Sync Tool pulls this data and transforms it to sync with other systems, such as ERP, PIM, or OMS. The Reconciliation Job regularly checks for missing data and pushes a synthetic webhook into the Queue to ensure proper syncing.
Cloud computing and data storage solutions such as message queues are recommended to handle bursts of traffic.
Novicell Middleware for Shopify
A few years ago, a team at Novicell embarked on a journey to try and accomplish just that - build a composable middleware for Shopify that could standardize integrations between various external systems and Shopify.
However, instead of building data-driven integrations, we take business requirements into account, create business-driven use cases and implement them. For example, instead of writing code that can “create a transaction of capture type for an amount of 100 kr.“, we write code that can “accept an invoice note from an external system and capture accepted card payment on the specified order“. In doing so, we can provide businesses with value without burdening them with the technical implementation details. Additionally, it allows us to change the implementation details in the future when there is a better way to do so with little to no impact on the business.
No business is the same, so we also add relevant extension points. This way, when we export orders from Shopify to clients’ systems, we can export exactly the data they need. Similarly, when we need to receive instructions from the ERP to capture order payments in Shopify, we define a business-driven (and Shopify-compatible) input, The ERP implementors can then choose to send data based on our data model or we can handle differences in the “adapter” component.
The below diagram illustrates the resulting mindset of aiming to build middleware solutions that could be distributed to each client with an “adapter“ - code that translates data coming from external systems to Shopify and vice-versa.
In doing so, we have been able to reap benefits both for the team and our clients:
- Reduced development time. Less code to write means we can finish in less time than before.
- Improved reliability. “Core“ components have a number of automated tests to try and catch errors early.
- Standardized documentation and training. Since middleware implementations based on our connectors share most of the same code, it allows us to write standardized documentation and onboard new employees more quickly across projects.
- Lower maintenance costs. Maintenance costs are typically lower because the “core“ component can be developed centrally and distributed to client implementation projects.
Complex integration processes are built and maintained in the green “Core“ component, while mapping and data exchange with specific client systems are handled in the blue “Adapter“ component.
We have organized our middleware into three separate connectors - one for products, orders, and customers. With such modularity, we can achieve greater flexibility in development as individual components can be updated or replaced without affecting the rest of the system. Additionally, we can handle scalability and reach performance requirements by scaling each connector. Moreover, Smaller, focused components are easier to troubleshoot, and maintain, as issues can be isolated to specific components.
The “Adapter“ component also acts as an anti-corruption layer, such that changes in the “core“ do not necessitate changes in the connection to the ERP system. For example, if we at a later time find out a better way to implement a particular use case, or if Shopify makes changes to their platform and APIs, we can make changes centrally in the connector(s) and distribute them to client’s middleware with minor or no changes required to the existing adapter code, meaning little to no work needs to be done on the ERP system.
Data flow
As mentioned earlier, we build our middleware connectors for Shopify based on the business needs of our clients. As a result, each connector implements a particular set of supported use cases that most of our clients share.
If a client requires particular customization, we can then investigate whether the functional requirement is something we should build into the connector, handle in the adapter component or build alongside the connectors in the middleware layer.
Product Connector
Middleware’s product connector primarily works with product and stock data. We build the adapter component to accept data from a range of different systems, sources, or a combination of them. Data is then mapped to our standardized connector input and sent to the connector. From there, pre-built, business-driven use cases are kicked off to orchestrate API calls to Shopify in the particular order they need to be made.
If product data is sent to the connector, but it has not changed since the last time data was sent, then no API requests are sent to Shopify. This is important, particularly with large catalogs, because if all data was being resent to Shopify over time, then it would take substantially longer to update all products and more importantly stock.
Another benefit is that the product connector can receive the product and stock information independently - we can stock information before we receive product information. Once we receive product information, we check if we already have received stock information earlier and, if we have, then we immediately update it in Shopify. It works the other way around too.
Order Connector
Middleware’s order connector primarily works with orders, transactions, and fulfillment. The adapter component is built for each client, allowing us to send and receive data from any ERP or other system they may have. The data we receive is mapped to a standard connector input and sent into the connector. At this point, business-driven use cases are run based on the input and orchestrate communications with Shopify.
When the order connector needs to send data back to the ERP or another system, it does so through the adapter component, where we specify what data from Shopify we need, how it should be formatted, and how it should be sent.
Communications with Shopify are done through the use of webhooks as well as reconciliation jobs to make sure data is up to date.
When exporting orders to external systems, we can prevent the same order from being sent multiple times. Additionally, through the adapter component, we can specify what validation rules an order must pass in Shopify before it is exported.
If the ERP system accidentally provides the same invoice note or a credit note, the order connector can prevent charging or refunding the customer too much by ignoring duplicate documents.
Finally, if the order connector is configured to talk to multiple Shopify stores, then we can automatically adjust inventory for order items in shops other than where the order was placed. This can lower the risk of overselling low-stock items before the ERP can respond with updated stock levels.
Customer Connector
Middleware’s customer connector primarily works with customer data. We build the adapter component to accept input from external systems. Data is mapped to our standardized connector input and sent to the connector. From there, pre-built, API calls to Shopify are orchestrated to create or update the customer, addresses, and custom fields.
When a customer is synchronized to Shopify, then any identifiable customer data in the connector is scrubbed.
What about low-code integration platforms?
While low-code integration platforms can be a quick and easy way to connect a range of systems, it's important to note that their built-in connectors may not always meet the specific business requirements of a particular company. Many of these connectors are designed for data-driven integrations and may not provide the level of customization needed to create a fully optimized workflow.
Furthermore, some low code integration platforms may rely on older technologies, such as Shopify's REST API, rather than the newer and more advanced GraphQL API. By using the REST API, businesses may miss out on the latest features and updates that are only available through the GraphQL API. This could potentially limit a company's ability to take advantage of Shopify's rapid pace of innovation and adapt to changing market trends.
In contrast, our connectors are designed with a deep understanding of Shopify's APIs and the specific needs of businesses. By leveraging the latest technologies and updates from Shopify, our connectors aim to provide up-to-date and optimized workflows.
On the otherhood, it is also possible to combine the best of both worlds - low code integration platform to handle enterprise-grad middleware layer with our connectors for integrating with Shopify. Instead of connectors “talking“ to the ERP integration points, it would communicate with the integration platform.
