InfluxDB 3.0: Flux Language Reintroduction Or Compatibility

Dear InfluxDB Team,

This article addresses a critical issue voiced by many community users who rely on the unique capabilities of the Flux language. We strongly urge you to reconsider or implement a compatibility layer for Flux within the InfluxDB 3.0 (Arrow/DataFusion-based IOx) architecture. The loss of Flux has presented challenges for users who have come to depend on its efficiency and elegance in handling specific time-series query patterns.

📉 The Challenge: Loss of Elegant Time-Series Querying

While the migration to SQL and the IOx architecture brings commendable performance gains, the removal of Flux has resulted in a significant loss of the most elegant and efficient way to handle specific, high-frequency time-series query patterns. This challenge impacts developers who have grown accustomed to Flux's intuitive syntax and powerful features, particularly when dealing with complex data manipulations and aggregations. The transition to SQL requires developers to adapt to a different paradigm, potentially leading to increased complexity and maintenance overhead. Therefore, reintroducing Flux or providing a seamless compatibility layer is crucial for maintaining developer productivity and ensuring a smooth transition to InfluxDB 3.0. The community's reliance on Flux stems from its ability to simplify intricate time-series operations, making it an indispensable tool for many users. Ignoring this dependency could lead to decreased adoption rates and user dissatisfaction. Consequently, addressing this challenge is not merely a technical issue but a critical factor in the long-term success and community support of InfluxDB 3.0. In essence, the challenge lies in bridging the gap between the performance benefits of the new architecture and the usability advantages of Flux, ensuring that users do not have to sacrifice one for the other. A balanced solution that incorporates the strengths of both Flux and SQL is essential for meeting the diverse needs of the InfluxDB community.

✅ Key Arguments for Reintroducing Flux

The complexity introduced by the SQL-only approach directly impacts developer efficiency and system maintainability. Here are key arguments for reintroducing Flux:

1. Elegant Multi-Field Latest Value Querying is Gone

• In Flux: Obtaining the latest non-null value and its corresponding independent timestamp for multiple fields requires only the simple |> last() function. The simplicity of this operation in Flux highlights its efficiency and ease of use, which are crucial for developers working with time-sensitive data. The ability to quickly retrieve the most recent data points across multiple fields without complex queries is a significant advantage of Flux.
• In 3.0 (LVC): The LVC (last_cache()) solution, while fast, requires prior configuration and enabling. This sacrifices the dynamic, flexible, and out-of-the-box advantage of Flux, unnecessarily adding maintenance overhead for what is a fundamental time-series query. The need for pre-configuration and enabling the LVC solution introduces additional steps and complexity, deviating from the dynamic and flexible nature of Flux. This added overhead can be a significant burden for developers, especially when dealing with rapidly changing data requirements. The simplicity and immediacy of Flux are lost, potentially leading to increased development time and effort. The contrast between Flux's straightforward approach and the LVC solution underscores the importance of retaining Flux's ease of use, which is vital for many time-series applications. Therefore, the argument for reintroducing Flux is not just about functionality but also about maintaining a user-friendly experience that empowers developers to work efficiently.

2. Superiority in Complex Data Flow and ETL

• Flux’s pipeline (|>) operations are ideal for time-series ETL, cross-measurement joins, customized data resampling, and complex aggregation. The pipeline operations in Flux enable a seamless and intuitive data transformation process, making it easier to handle complex time-series data manipulations. The ability to chain operations together in a clear and concise manner is a significant advantage of Flux, particularly when dealing with intricate data flows. These features are essential for tasks such as data cleaning, transformation, and loading, which are critical components of any data-driven application. The ease with which Flux handles cross-measurement joins and customized data resampling sets it apart from SQL, which often requires more verbose and complex queries to achieve the same results. The power and flexibility of Flux's ETL capabilities make it an invaluable tool for developers working with time-series data.
• Replicating this logic using standard SQL often necessitates verbose and error-prone subqueries, CTEs, and complex window functions, making the code less readable and harder to maintain than its Flux counterpart. The verbosity and complexity of SQL when replicating Flux's logic can lead to increased development time and a higher risk of errors. Subqueries, Common Table Expressions (CTEs), and complex window functions, while powerful, can make SQL queries difficult to understand and maintain. This complexity can be a significant obstacle for developers, particularly those who are new to time-series data processing. Flux's straightforward syntax and pipeline operations provide a more intuitive and efficient way to handle complex data transformations. The readability and maintainability of code are crucial for long-term project success, and Flux's superior handling of complex data flows makes it a valuable asset. By reintroducing Flux, InfluxDB 3.0 can ensure that developers have access to a language that simplifies intricate data manipulations, reducing the burden of writing and maintaining complex SQL queries. This ultimately leads to faster development cycles and more robust applications.

3. Community Investment and Migration Costs

• The community has invested substantial time and resources into learning Flux and building mission-critical workflows. The extensive investment of the community in learning Flux and developing critical workflows underscores the importance of considering the impact of its removal. Many users have dedicated significant time and effort to mastering Flux, and their expertise is a valuable asset. The codebases and workflows built using Flux represent a substantial investment, and the deprecation of Flux can render these investments obsolete. This can create a sense of frustration and loss within the community, as users are forced to abandon their existing knowledge and resources. Recognizing and respecting this investment is crucial for maintaining the trust and support of the community. A solution that allows users to leverage their existing Flux knowledge and codebases would be highly beneficial, mitigating the disruption caused by the transition to InfluxDB 3.0. This could involve providing a migration path or ensuring compatibility with existing Flux scripts.
• The deprecation forces users to abandon these existing codebases and knowledge assets, creating a significant barrier and cost for upgrading to InfluxDB 3.0. The need to rewrite existing codebases and re-learn query paradigms can be a significant barrier to adoption for InfluxDB 3.0. The time and resources required for this transition represent a substantial cost for users, potentially outweighing the benefits of the new architecture. This cost can be particularly burdensome for organizations that have built their entire data infrastructure around Flux. The loss of knowledge assets further exacerbates the challenge, as developers need to acquire new skills and expertise to work with SQL-based time-series queries. This learning curve can be steep, and the resulting disruption can impact productivity and project timelines. Therefore, providing a smooth migration path or a compatibility layer for Flux is essential for minimizing the cost and disruption associated with upgrading to InfluxDB 3.0. By addressing these concerns, InfluxDB can ensure that the transition is as seamless as possible for its users, fostering continued adoption and support for the platform.

💡 Proposed Solutions (Recommendations for the Roadmap)

We understand the engineering challenge of running the native Flux engine atop DataFusion. We propose the team evaluate the feasibility of the following alternatives to restore this vital functionality:

1. Develop a Flux-to-SQL Compiler: Offer a compatibility layer that efficiently translates common Flux patterns (e.g., |> last(), |> group(), |> aggregateWindow()) into highly optimized DataFusion SQL or Arrow native operations. A Flux-to-SQL compiler would provide a seamless transition for users by allowing them to continue using Flux while leveraging the performance benefits of DataFusion SQL. This compatibility layer would translate common Flux patterns into optimized SQL queries, ensuring that users do not have to rewrite their existing codebases. The compiler would need to be efficient and accurate, ensuring that the translated SQL queries maintain the functionality and performance of the original Flux code. This approach would strike a balance between preserving the familiarity of Flux and leveraging the underlying SQL engine, making it an attractive option for many users. The development of such a compiler would require a deep understanding of both Flux and SQL, as well as the optimization techniques necessary to ensure efficient query execution. However, the benefits of providing a smooth migration path and preserving user investments in Flux make this a worthwhile endeavor. By implementing a Flux-to-SQL compiler, InfluxDB can maintain the continuity of its user experience while advancing its technology.
2. Reintroduce Core Flux Functions as UDFs: At the very least, re-introduce the behavior of Flux functions that are notoriously difficult to replicate elegantly in standard SQL (e.g., the true multi-series latest value behavior of last()), as specialized Time-Series User-Defined Functions (UDFs). Reintroducing core Flux functions as User-Defined Functions (UDFs) in SQL would address the specific challenges users face when trying to replicate Flux's functionality in SQL. Certain Flux functions, such as last(), are particularly difficult to implement elegantly in SQL, and their absence can significantly impact the usability of InfluxDB 3.0. By providing these functions as UDFs, InfluxDB can offer a practical solution that bridges the gap between Flux and SQL. UDFs allow users to extend the functionality of SQL with custom functions, providing a way to incorporate Flux's capabilities into the SQL environment. This approach would enable users to perform complex time-series operations with greater ease and efficiency. The implementation of UDFs would require careful consideration to ensure that they are well-integrated into the SQL engine and perform optimally. However, the benefits of providing a more comprehensive set of time-series functions in SQL make this a valuable addition to InfluxDB 3.0. By reintroducing core Flux functions as UDFs, InfluxDB can enhance the user experience and ensure that developers have the tools they need to work effectively with time-series data.

We sincerely hope the InfluxDB team considers this critical community feedback and includes a path for feature parity or function migration for existing Flux users in the future roadmap.

Thank you for your time and attention to this matter.

For more information about InfluxDB and time-series databases, you can visit the official InfluxData website: InfluxData