Sparklens is an open-source framework to improve the performance of Spark applications. Sparklens can inspect a Spark application and recommend the optimal configuration, such as the number of executors or cores to use.
RubiX is a lightweight data caching framework that can be used by big data engines such as Hadoop. Rubix uses local disks to provide the best I/O bandwidth to the big data engines. Rubix is useful in shared storage architectures where the data execution engine is separate from storage. For example, on public clouds like AWS or Microsoft Azure, data is stored in cloud storage and the engine accesses the data over a network. Similarly, in data centers, Presto runs on a separate cluster from HDFS and accesses data over the network.
RubiX can be extended to support any engine that accesses data using the Hadoop FileSystem interface via plugins. There are plugins to access data on AWS S3, Microsoft Azure Blob Store, and HDFS. RubiX can be extended for use with any other storage system including other cloud stores.
Free access to Qubole for 30 days to build data pipelines, bring machine learning to production, and analyze any data type from any data source.
See what our Open Data Lake Platform can do for you in 35 minutes.