Analysing and comparing open source software requires the adoption of quality models that have to be evaluated in order to rank the software systems being compared and to support the final decision about which of them has to be adopted. Since software quality can mean different things in different scenarios, quality models should be flexible in order to accommodate the needs of different users. Over the years several quality models have been proposed. Even though some of them are tool supported, they cannot be extended or customized to better accommodate the requirements of specific business contexts. In this paper, instead of having a fixed model, we propose a workflow and a tool chain to sup- port the specification of custom quality models, which can guide the automated analysis of open source software.