Using Shiny and Rmarkdown for Rapid Delivery of Analytics and CRO Reports
Optimized analytics and reporting pipeline
Our main goal at Altamedia is to deliver high-quality analytics and recommendations to our valued customers as fast as possible. Our team encompasses a varied set of expertise, from UI/UX researchers and CRO experts, through data scientists to database engineers. All this with a good purpose: we optimize our workflows by aligning tasks and expertise, thereby minimizing time to delivery and maximizing throughput.
As part of this strategy, we try, to the degree possible, to decouple the research and analytics (data wrangling, exploratory data analysis, charting, modelling, write-up) from the reporting (creating interactive graphics, developing and deploying dashboards). By doing so, on one hand, we liberate the scientist to focus on understanding the data and converting it to meaningful information for the client, and, on the other, we give time to the dashboard engineer to focus on stable and scalable reporting products. All too often we see the problems when a statistician is expected to miraculously develop a dashboard, and vice versa, when a software developer is suddenly supposed to perform some urgent time series forecast.
To make this work, we have been focusing on a system for semi-automated delivery of analytics/CRO reports. The main goal we set for ourselves was to enable the CRO experts “one-step publishing”. In other words, after having analyzed the data and written the report, the researcher, and company as a whole, spends minimal time to deliver insight and recommendations to the client. In turn, the reports that provide long-term value for the client, are then converted into dashboard products by our engineering team such that the client can derive continuous value.
Below, we briefly describe our system from a technical standpoint.
Enter Shiny and Rmarkdown
In a nutshell, our approach can be abstracted to two main steps:
Researchers prepare Rmarkdown reports and upload them to a cloud location
Engineers develop a Shiny application that links to the cloud location and automatically serves these Rmarkdown reports
We will not spend much time to discuss the Rmarkdown reports. These can vary widely in content and length, and are geared towards presenting highlights, metrics and charts that abstract the complexity and messiness of the underlying data into insights for the client. From a technical perspective, as long as the software dependencies are satisfied, these reports would all be automatically rendered when the client visits the rapid reporting portal.
Instead, we’ll spend a bit of time discussing our Shiny set-up. As with most of our Shiny work, we employ the golem framework to standardize and streamline our application development process. We deploy to various outlets, depending on the client requirements, but always maintaining the highest security and privacy standards.
First, our cloud location directory structure. Each rapid Rmarkdown report is placed in a corresponding folder with an informative name. The folder name would later become the page name in the Shiny portal, while the contained Rmarkdown would be rendered to HTML and displayed as contents of the automatically-generated page. Then, the Shiny portal server, consumes the contents of the rapid reporting directory, and populates the dashboard in two steps:
Cycle over the sub-directories to generate website pages for each report. Each page additionally has three buttons to download each report as an HTML, PDF, or MS Word document.
Cycle over the Rmarkdown files within the folder to knit each report and convert to HTML that can be included in Shiny. The code also notifies the user about the rendering progress, which depending on the amount of data being processed might take a few seconds. Finally, as we create the content for each page, we also include the requisite code to handle downloads so that the user can save the report.
And that is more-or-less it, as far as the crux of our rapid reporting Shiny setup is concerned. Of course, the application has a few more layers and additional styling and customization for our clients, but the basics are covered by the server logic presented above.
Summary
Our system leverages the great resources available in the R/Shiny/Rmarkdown environment to quickly and efficiently deliver results to our clients. From a technical design standpoint in this method, our goal is not to build a comprehensive static web portal, rather to communicate our findings with the clients as fast as possible. Thereafter, we make joint decisions about analyses and metrics that need to be included in a long-term, continuously-updated reporting platform that we curate and maintain for each of our clients.
