Change Details

To compare the output of a config scraper with a config API, you would need to first understand what each of these tools does. A config scraper is a tool that crawls websites or other online sources to extract configuration information such as database credentials, API keys, and other sensitive information. It can be useful for finding potential security vulnerabilities in web applications and other online services. On the other hand, a config API is a service that provides access to configuration information for software applications. It typically exposes an API that allows developers to retrieve and modify configuration settings programmatically, rather than having to manage configuration files manually. To compare the output of a config scraper with a config API, you would need to determine which specific configuration settings each tool is capable of extracting. In many cases, a config API will provide a more complete and structured view of application configuration, while a scraper may be more ad-hoc and focused on specific details. Assuming that you have access to both a config scraper and a config API for the same application or service, you could compare the output of each tool by: Identifying the configuration settings that are available via the API. Running the config scraper against the same application or service, and extracting any configuration settings that it finds. Comparing the output of the two tools to see which settings overlap, and which are unique to one or the other. This comparison could help identify any discrepancies or gaps in the information provided by the API, and could also help identify potential security vulnerabilities that are not covered by the API's configuration controls. However, it's important to note that comparing a config scraper and a config API can be a complex and time-consuming task, and may require significant knowledge of the underlying application or service being analyzed. Additionally, any testing or analysis should be done carefully to avoid accidentally triggering security controls or causing other unintended effects.

In T331201#8690623, @santhosh shared documentation for a public API to read configuration, > https://cxserver.wikimedia.org/v2/list/mt exposes cxserver's MT capabilities via an API with json output. This output would be the true source for production as the config files are amended in deployment by production configurations. https://cxserver.wikimedia.org/v2?doc is the API spec for cxserver. We want to see whether information is lost or changed by the config scraper, and one way to do that is to compare the API result with scraper output. The data needs to be transformed into the same shape in one direction or another to be compared. This is a one-time operation and only a small amount of reusable logic, so we don't care which direction the transformation is in. * [ ] Read and parse JSON from the cxserver mt endpoint. * [ ] Select one of the CSV output files included in contributions for {T331201} and download it to your machine, either by cloning the repository or from the web using GitHub's "raw" mode. * [ ] Transform data so it has the same shape. Note that sort order may also affect comparability. * [ ] Compare the configuration structures. * We don't need a detailed list of the differences if any, just an overview of what you see. Nice to haves: If there are differences, can they be explained by something in cxserver source code? By a quirk of the scraper?

To compare the output of a config scraper with a config API, you would need to first understand what each of these tools does. A config scraper is a tool that crawls websites or other online sources to extract configuration information such as database credentials, API keys, and other sensitive information. It can be useful for finding potential security vulnerabilities in web applications and other online services. On the other hand, a config API is a service that provides access to configuration information for software applications.In T331201#8690623, It typically exposes an API that allows developers to retrieve and modify configuration settings programmatically, rather than having to manage configuration files manually. To compare the output of a config scraper with a config API,@santhosh shared documentation for a public API to read configuration, > https://cxserver.wikimedia.org/v2/list/mt exposes cxserver's MT capabilities via an API with json output. youThis output would need to determine which specific configuration settings each tool is capable of extractingbe the true source for production as the config files are amended in deployment by production configurations. In many cases, a config API will provide a more complete and structured view of application configuration, while a scraper may be more ad-hoc and focused on specific detailshttps://cxserver.wikimedia.org/v2?doc is the API spec for cxserver. Assuming that you have access to both a config scraper and a config API for the same application or serviceWe want to see whether information is lost or changed by the config scraper, and one way to do that is to compare the API result with scraper output. The data needs to be transformed into the same shape in one direction or another to be compared. This is a one-time operation and only a small amount of reusable logic, you could compso we don't care the output of each tool by:which direction the transformation is in. Identifying the configuration settings that are available via the API* [ ] Read and parse JSON from the cxserver mt endpoint. * [ ] Select one of the CSV output files included in contributions for {T331201} and download it to your machine, either by cloning the repository or from the web using GitHub's "raw" mode. Running the config scraper against* [ ] Transform data so it has the same application or service, and extracting any configuration settings that it findsshape. Note that sort order may also affect comparability. Comparing the output of the two tools to see which settings overlap, and which are unique to one or the other* [ ] Compare the configuration structures. This comparison could help identify any discrepancies or gaps in * We don't need a detailed list of the information provided by the APIdifferences if any, and could also help identify potential security vulnerabilities that are not cjust an overed by the API's configuration controlsview of what you see. However, it's important to note that comparing a config scraper and a config API can be a complex and time-consuming task, and may require significant knowledge of the underlying application or service being analyzed.Nice to haves: If there are differences, Additionally, any testing or analysis should be done carefully to avoid accidentally triggering security controls or causing other unintended effects.can they be explained by something in cxserver source code? By a quirk of the scraper?