Eintrag weiter verarbeiten
Using DODAS as deployment manager for smart caching of CMS data management system
Gespeichert in:
| Zeitschriftentitel: | Journal of Physics: Conference Series |
|---|---|
| Personen und Körperschaften: | , , , , , , , , , , |
| In: | Journal of Physics: Conference Series, 1525, 2020, 1, S. 012057 |
| Format: | E-Article |
| Sprache: | Unbestimmt |
| veröffentlicht: |
IOP Publishing
|
| Schlagwörter: |
| author_facet |
Tracolli, M. Antonacci, M. Boccali, T. Bonacorsi, D. Ciangottini, D Donvito, G. Duma, C. Gaido, L. Salomoni, D. Spiga, D. Kuznetsov, V. Tracolli, M. Antonacci, M. Boccali, T. Bonacorsi, D. Ciangottini, D Donvito, G. Duma, C. Gaido, L. Salomoni, D. Spiga, D. Kuznetsov, V. |
|---|---|
| author |
Tracolli, M. Antonacci, M. Boccali, T. Bonacorsi, D. Ciangottini, D Donvito, G. Duma, C. Gaido, L. Salomoni, D. Spiga, D. Kuznetsov, V. |
| spellingShingle |
Tracolli, M. Antonacci, M. Boccali, T. Bonacorsi, D. Ciangottini, D Donvito, G. Duma, C. Gaido, L. Salomoni, D. Spiga, D. Kuznetsov, V. Journal of Physics: Conference Series Using DODAS as deployment manager for smart caching of CMS data management system General Physics and Astronomy |
| author_sort |
tracolli, m. |
| spelling |
Tracolli, M. Antonacci, M. Boccali, T. Bonacorsi, D. Ciangottini, D Donvito, G. Duma, C. Gaido, L. Salomoni, D. Spiga, D. Kuznetsov, V. 1742-6588 1742-6596 IOP Publishing General Physics and Astronomy http://dx.doi.org/10.1088/1742-6596/1525/1/012057 <jats:title>Abstract</jats:title> <jats:p>DODAS stands for Dynamic On Demand Analysis Service and is a Platform as a Service toolkit built around several EOSC-hub services designed to instantiate and configure on-demand container-based clusters over public or private Cloud resources. It automates the whole workflow from service provisioning to the configuration and setup of software applications. Therefore, such a solution allows using “any cloud provider”, with almost zero effort. In this paper, we demonstrate how DODAS can be adopted as a deployment manager to set up and manage the compute resources and services required to develop an AI solution for smart data caching. The smart caching layer may reduce the operational cost and increase flexibility with respect to regular centrally managed storage of the current CMS computing model. The cache space should be dynamically populated with the most requested data. In addition, clustering such caching systems will allow to operate them as a Content Delivery System between data providers and end-users. Moreover, a geographically distributed caching layer will be functional also to a data-lake based model, where many satellite computing centers might appear and disappear dynamically. In this context, our strategy is to develop a flexible and automated AI environment for smart management of the content of such clustered cache system. In this contribution, we will describe the identified computational phases required for the AI environment implementation, as well as the related DODAS integration. Therefore we will start with the overview of the architecture for the pre-processing step, based on Spark, which has the role to prepare data for a Machine Learning technique. A focus will be given on the automation implemented through DODAS. Then, we will show how to train an AI-based smart cache and how we implemented a training facility managed through DODAS. Finally, we provide an overview of the inference system, based on the CMS-TensorFlow as a Service and also deployed as a DODAS service.</jats:p> Using DODAS as deployment manager for smart caching of CMS data management system Journal of Physics: Conference Series |
| doi_str_mv |
10.1088/1742-6596/1525/1/012057 |
| facet_avail |
Online Free |
| format |
ElectronicArticle |
| fullrecord |
blob:ai-49-aHR0cDovL2R4LmRvaS5vcmcvMTAuMTA4OC8xNzQyLTY1OTYvMTUyNS8xLzAxMjA1Nw |
| id |
ai-49-aHR0cDovL2R4LmRvaS5vcmcvMTAuMTA4OC8xNzQyLTY1OTYvMTUyNS8xLzAxMjA1Nw |
| institution |
DE-L229 DE-D275 DE-Bn3 DE-Brt1 DE-Zwi2 DE-D161 DE-Gla1 DE-Zi4 DE-15 DE-Pl11 DE-Rs1 DE-105 DE-14 DE-Ch1 |
| imprint |
IOP Publishing, 2020 |
| imprint_str_mv |
IOP Publishing, 2020 |
| issn |
1742-6588 1742-6596 |
| issn_str_mv |
1742-6588 1742-6596 |
| language |
Undetermined |
| mega_collection |
IOP Publishing (CrossRef) |
| match_str |
tracolli2020usingdodasasdeploymentmanagerforsmartcachingofcmsdatamanagementsystem |
| publishDateSort |
2020 |
| publisher |
IOP Publishing |
| recordtype |
ai |
| record_format |
ai |
| series |
Journal of Physics: Conference Series |
| source_id |
49 |
| title |
Using DODAS as deployment manager for smart caching of CMS data management system |
| title_unstemmed |
Using DODAS as deployment manager for smart caching of CMS data management system |
| title_full |
Using DODAS as deployment manager for smart caching of CMS data management system |
| title_fullStr |
Using DODAS as deployment manager for smart caching of CMS data management system |
| title_full_unstemmed |
Using DODAS as deployment manager for smart caching of CMS data management system |
| title_short |
Using DODAS as deployment manager for smart caching of CMS data management system |
| title_sort |
using dodas as deployment manager for smart caching of cms data management system |
| topic |
General Physics and Astronomy |
| url |
http://dx.doi.org/10.1088/1742-6596/1525/1/012057 |
| publishDate |
2020 |
| physical |
012057 |
| description |
<jats:title>Abstract</jats:title>
<jats:p>DODAS stands for Dynamic On Demand Analysis Service and is a Platform as a Service toolkit built around several EOSC-hub services designed to instantiate and configure on-demand container-based clusters over public or private Cloud resources. It automates the whole workflow from service provisioning to the configuration and setup of software applications. Therefore, such a solution allows using “any cloud provider”, with almost zero effort. In this paper, we demonstrate how DODAS can be adopted as a deployment manager to set up and manage the compute resources and services required to develop an AI solution for smart data caching. The smart caching layer may reduce the operational cost and increase flexibility with respect to regular centrally managed storage of the current CMS computing model. The cache space should be dynamically populated with the most requested data. In addition, clustering such caching systems will allow to operate them as a Content Delivery System between data providers and end-users. Moreover, a geographically distributed caching layer will be functional also to a data-lake based model, where many satellite computing centers might appear and disappear dynamically. In this context, our strategy is to develop a flexible and automated AI environment for smart management of the content of such clustered cache system. In this contribution, we will describe the identified computational phases required for the AI environment implementation, as well as the related DODAS integration. Therefore we will start with the overview of the architecture for the pre-processing step, based on Spark, which has the role to prepare data for a Machine Learning technique. A focus will be given on the automation implemented through DODAS. Then, we will show how to train an AI-based smart cache and how we implemented a training facility managed through DODAS. Finally, we provide an overview of the inference system, based on the CMS-TensorFlow as a Service and also deployed as a DODAS service.</jats:p> |
| container_issue |
1 |
| container_start_page |
0 |
| container_title |
Journal of Physics: Conference Series |
| container_volume |
1525 |
| format_de105 |
Article, E-Article |
| format_de14 |
Article, E-Article |
| format_de15 |
Article, E-Article |
| format_de520 |
Article, E-Article |
| format_de540 |
Article, E-Article |
| format_dech1 |
Article, E-Article |
| format_ded117 |
Article, E-Article |
| format_degla1 |
E-Article |
| format_del152 |
Buch |
| format_del189 |
Article, E-Article |
| format_dezi4 |
Article |
| format_dezwi2 |
Article, E-Article |
| format_finc |
Article, E-Article |
| format_nrw |
Article, E-Article |
| _version_ |
1792326456498454536 |
| geogr_code |
not assigned |
| last_indexed |
2024-03-01T12:21:39.509Z |
| geogr_code_person |
not assigned |
| openURL |
url_ver=Z39.88-2004&ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fvufind.svn.sourceforge.net%3Agenerator&rft.title=Using+DODAS+as+deployment+manager+for+smart+caching+of+CMS+data+management+system&rft.date=2020-04-01&genre=article&issn=1742-6596&volume=1525&issue=1&pages=012057&jtitle=Journal+of+Physics%3A+Conference+Series&atitle=Using+DODAS+as+deployment+manager+for+smart+caching+of+CMS+data+management+system&aulast=Kuznetsov&aufirst=V.&rft_id=info%3Adoi%2F10.1088%2F1742-6596%2F1525%2F1%2F012057&rft.language%5B0%5D=und |
| SOLR | |
| _version_ | 1792326456498454536 |
| author | Tracolli, M., Antonacci, M., Boccali, T., Bonacorsi, D., Ciangottini, D, Donvito, G., Duma, C., Gaido, L., Salomoni, D., Spiga, D., Kuznetsov, V. |
| author_facet | Tracolli, M., Antonacci, M., Boccali, T., Bonacorsi, D., Ciangottini, D, Donvito, G., Duma, C., Gaido, L., Salomoni, D., Spiga, D., Kuznetsov, V., Tracolli, M., Antonacci, M., Boccali, T., Bonacorsi, D., Ciangottini, D, Donvito, G., Duma, C., Gaido, L., Salomoni, D., Spiga, D., Kuznetsov, V. |
| author_sort | tracolli, m. |
| container_issue | 1 |
| container_start_page | 0 |
| container_title | Journal of Physics: Conference Series |
| container_volume | 1525 |
| description | <jats:title>Abstract</jats:title> <jats:p>DODAS stands for Dynamic On Demand Analysis Service and is a Platform as a Service toolkit built around several EOSC-hub services designed to instantiate and configure on-demand container-based clusters over public or private Cloud resources. It automates the whole workflow from service provisioning to the configuration and setup of software applications. Therefore, such a solution allows using “any cloud provider”, with almost zero effort. In this paper, we demonstrate how DODAS can be adopted as a deployment manager to set up and manage the compute resources and services required to develop an AI solution for smart data caching. The smart caching layer may reduce the operational cost and increase flexibility with respect to regular centrally managed storage of the current CMS computing model. The cache space should be dynamically populated with the most requested data. In addition, clustering such caching systems will allow to operate them as a Content Delivery System between data providers and end-users. Moreover, a geographically distributed caching layer will be functional also to a data-lake based model, where many satellite computing centers might appear and disappear dynamically. In this context, our strategy is to develop a flexible and automated AI environment for smart management of the content of such clustered cache system. In this contribution, we will describe the identified computational phases required for the AI environment implementation, as well as the related DODAS integration. Therefore we will start with the overview of the architecture for the pre-processing step, based on Spark, which has the role to prepare data for a Machine Learning technique. A focus will be given on the automation implemented through DODAS. Then, we will show how to train an AI-based smart cache and how we implemented a training facility managed through DODAS. Finally, we provide an overview of the inference system, based on the CMS-TensorFlow as a Service and also deployed as a DODAS service.</jats:p> |
| doi_str_mv | 10.1088/1742-6596/1525/1/012057 |
| facet_avail | Online, Free |
| format | ElectronicArticle |
| format_de105 | Article, E-Article |
| format_de14 | Article, E-Article |
| format_de15 | Article, E-Article |
| format_de520 | Article, E-Article |
| format_de540 | Article, E-Article |
| format_dech1 | Article, E-Article |
| format_ded117 | Article, E-Article |
| format_degla1 | E-Article |
| format_del152 | Buch |
| format_del189 | Article, E-Article |
| format_dezi4 | Article |
| format_dezwi2 | Article, E-Article |
| format_finc | Article, E-Article |
| format_nrw | Article, E-Article |
| geogr_code | not assigned |
| geogr_code_person | not assigned |
| id | ai-49-aHR0cDovL2R4LmRvaS5vcmcvMTAuMTA4OC8xNzQyLTY1OTYvMTUyNS8xLzAxMjA1Nw |
| imprint | IOP Publishing, 2020 |
| imprint_str_mv | IOP Publishing, 2020 |
| institution | DE-L229, DE-D275, DE-Bn3, DE-Brt1, DE-Zwi2, DE-D161, DE-Gla1, DE-Zi4, DE-15, DE-Pl11, DE-Rs1, DE-105, DE-14, DE-Ch1 |
| issn | 1742-6588, 1742-6596 |
| issn_str_mv | 1742-6588, 1742-6596 |
| language | Undetermined |
| last_indexed | 2024-03-01T12:21:39.509Z |
| match_str | tracolli2020usingdodasasdeploymentmanagerforsmartcachingofcmsdatamanagementsystem |
| mega_collection | IOP Publishing (CrossRef) |
| physical | 012057 |
| publishDate | 2020 |
| publishDateSort | 2020 |
| publisher | IOP Publishing |
| record_format | ai |
| recordtype | ai |
| series | Journal of Physics: Conference Series |
| source_id | 49 |
| spelling | Tracolli, M. Antonacci, M. Boccali, T. Bonacorsi, D. Ciangottini, D Donvito, G. Duma, C. Gaido, L. Salomoni, D. Spiga, D. Kuznetsov, V. 1742-6588 1742-6596 IOP Publishing General Physics and Astronomy http://dx.doi.org/10.1088/1742-6596/1525/1/012057 <jats:title>Abstract</jats:title> <jats:p>DODAS stands for Dynamic On Demand Analysis Service and is a Platform as a Service toolkit built around several EOSC-hub services designed to instantiate and configure on-demand container-based clusters over public or private Cloud resources. It automates the whole workflow from service provisioning to the configuration and setup of software applications. Therefore, such a solution allows using “any cloud provider”, with almost zero effort. In this paper, we demonstrate how DODAS can be adopted as a deployment manager to set up and manage the compute resources and services required to develop an AI solution for smart data caching. The smart caching layer may reduce the operational cost and increase flexibility with respect to regular centrally managed storage of the current CMS computing model. The cache space should be dynamically populated with the most requested data. In addition, clustering such caching systems will allow to operate them as a Content Delivery System between data providers and end-users. Moreover, a geographically distributed caching layer will be functional also to a data-lake based model, where many satellite computing centers might appear and disappear dynamically. In this context, our strategy is to develop a flexible and automated AI environment for smart management of the content of such clustered cache system. In this contribution, we will describe the identified computational phases required for the AI environment implementation, as well as the related DODAS integration. Therefore we will start with the overview of the architecture for the pre-processing step, based on Spark, which has the role to prepare data for a Machine Learning technique. A focus will be given on the automation implemented through DODAS. Then, we will show how to train an AI-based smart cache and how we implemented a training facility managed through DODAS. Finally, we provide an overview of the inference system, based on the CMS-TensorFlow as a Service and also deployed as a DODAS service.</jats:p> Using DODAS as deployment manager for smart caching of CMS data management system Journal of Physics: Conference Series |
| spellingShingle | Tracolli, M., Antonacci, M., Boccali, T., Bonacorsi, D., Ciangottini, D, Donvito, G., Duma, C., Gaido, L., Salomoni, D., Spiga, D., Kuznetsov, V., Journal of Physics: Conference Series, Using DODAS as deployment manager for smart caching of CMS data management system, General Physics and Astronomy |
| title | Using DODAS as deployment manager for smart caching of CMS data management system |
| title_full | Using DODAS as deployment manager for smart caching of CMS data management system |
| title_fullStr | Using DODAS as deployment manager for smart caching of CMS data management system |
| title_full_unstemmed | Using DODAS as deployment manager for smart caching of CMS data management system |
| title_short | Using DODAS as deployment manager for smart caching of CMS data management system |
| title_sort | using dodas as deployment manager for smart caching of cms data management system |
| title_unstemmed | Using DODAS as deployment manager for smart caching of CMS data management system |
| topic | General Physics and Astronomy |
| url | http://dx.doi.org/10.1088/1742-6596/1525/1/012057 |