MSG SERV CLM数据集下载

© 2023-2024, Jiheng Hu. 原创内容，禁止转载。

Meteosat Second Generation

EUMETSAT第二代静止卫星系列包含两个卫星点位，分别是位于欧洲-非洲上方的0-degree点位，由在运行的METSAT-10和11业务覆盖；位于印度洋上空的Indian Ocean点位，先后由METSAT-8(41.5°E)和METSAT-9(45.5°E)业务覆盖。
这两点位的卫星视角如下图：

MSG视角: 0-Degree(左)，Indian Ocean(右)

MSG的Cloud Mask产品主要有两个：Cloud Mask - MSG - 0 degree和Cloud Mask - MSG - Indian Ocean.其时间范围分别为 2004年01月至今和2017年1月至今。
该数据集是MSG搭载的SEVI中分辨率成像仪的全盘扫描结果，时间间隔为15分钟。

考虑到个人需求，本次要下载的是Cloud Mask - MSG - Indian Ocean产品。该数据集的主要参数如下：

Cloud Mask - MSG - Indian Ocean

• Status: Operational
• Temporal extent : 31/01/2017 to now
• Processing level : Level 2 Data
• Region:
  • Latitude : -67.5 to 67.5 degrees
  • Longitude : -22 to 113 degrees
  • Resolution: 0100*0100
  • Interval: 15 min

Cloud Mask下载

数据集的详情页提供了下载地址：https://data.eumetsat.int/data/map/EO:EUM:DAT:MSG:CLM
我们可以输入时间范围进行文件筛选，订单下载或者下载单个文件。

EUMSAT DATA SERVICE

这里文件的详情可以看出，该数据集的单次扫描的文件为如下的zip格式打包文件，内部包含所需要的观测文件，格式是grib2：

+ MSG3-SEVI-MSGCLMK-0100-0100-20231214060000.000000000Z-NA.zip
 - MSG3-SEVI-MSGCLMK-0100-0100-20231214060000.000000000Z-NA.grb
 - metadata.xml
 - manifest.xml

调用API下载

考虑到该数据的文件名较为规律，可以通过调用API的方式直接从数据仓库中下载需要的数据。
点击右下角的API block可以发现有数据中心提供了好几个用于后台调用下载的API服务。

1. 注册网站，获取API Key。
   点击右上角的login, 如果没有账号就注册并登录。
   按照下图或者去api-key 页面复制你的API Key和secret,这两者包含了你的账户信息和验证信息。
   

   GET YOUR API KEY
   
   页面下方的TOKEN不需要复制，系统会生成一小时时效的TOKEN,只要实时获取就可以了，后面的代码会实现验证功能。

2. 安装emudac库
   参考Get EUMDAC，这里可以直接使用bash安装或者python安装
   考虑到计划使用python脚本进行下载，这里使用pip安装方式，注意pip版本必须>3.7才能安装。

   > pip install eumdac

3. 编写代码
   这里可以参考Jupyter+Notebook+Using+the+Download+API进行脚本的编写，也可以改写成python脚本。

• 首先引入必要的库，粘贴你刚才复制的API Key和Secret到对应的字符串。系统会自动验证身份并返回TOKEN,一般一个小时内有效。通过这种方式可以总是获取有效的TOKEN。

  import eumdac     ## pip3 istall eumdac
  import datetime
  import shutil
  import requests
  import time
  
  # Insert your personal key and secret into the single quotes
  
  consumer_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa'
  consumer_secret = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa'
  
  credentials = (consumer_key, consumer_secret)
  
  token = eumdac.AccessToken(credentials)
  
  try:
      print(f"This token '{token}' expires {token.expiration}")
  except requests.exceptions.HTTPError as error:
      print(f"Unexpected error: {error}")

  [OUT] This token ‘f88198d1-d2e5-33ae-b61e-4f7176e12941’ expires 2023-12-02 12:31:46.611048

• 按照产品编号EO:EUM:DAT:MSG:CLM-IODC来查询数据集，
  Let’s start to select a collection we want to browse through. For more information on determining which collections are available, see the previous tutorial, Discovering collections

  datastore = eumdac.DataStore(token)
  
  try:    
      # https://data.eumetsat.int/product/EO:EUM:DAT:MSG:CLM-IODC 
  
      selected_collection = datastore.get_collection('EO:EUM:DAT:MSG:CLM-IODC') 
      print(f"{selected_collection} - {selected_collection.title}")
  except eumdac.datastore.DataStoreError as error:
      print(f"Error related to the data store: '{error.msg}'")
  except eumdac.collection.CollectionError as error:
      print(f"Error related to the collection: '{error.msg}'")
  except requests.exceptions.ConnectionError as error:
      print(f"Error related to the connection: '{error.msg}'")
  except requests.exceptions.RequestException as error:
      print(f"Unexpected error: {error}")

• 设置起始结束时间来查询

  # Set sensing start and end time
  start = datetime.datetime(2021, 11, 10, 8, 0)
  end = datetime.datetime(2021, 11, 10, 9, 0)
  
  # Retrieve datasets that match our filter
  products = selected_collection.search(
      dtstart=start,
      dtend=end)
  
  for product in products:
      try:
          print(product)
      except eumdac.collection.CollectionError as error:
          print(f"Error related to the collection: '{error.msg}'")
      except requests.exceptions.ConnectionError as error:
          print(f"Error related to the connection: '{error.msg}'")
      except requests.exceptions.RequestException as error:
          print(f"Unexpected error: {error}")

• 直接下载查询到的所有文件

  ###  Download all products from search
  for product in products:
      try:
          with product.open() as fsrc, \
                  open(fsrc.name, mode='wb') as fdst:
              shutil.copyfileobj(fsrc, fdst)
              print(f'Download of product {product} finished.')
      except eumdac.product.ProductError as error:
          print(f"Error related to the product '{product}' while trying to download it: '{error.msg}'")
      except requests.exceptions.ConnectionError as error:
          print(f"Error related to the connection: '{error.msg}'")
      except requests.exceptions.RequestException as error:
          print(f"Unexpected error: {error}")
      
  print('All downloads are finished.')

• 或者指定文件名下载单个文件

  ## Download a single product
  
  selected_product = datastore.get_product(
      product_id='MSG2-SEVI-MSGCLMK-0100-0100-20231104141500.000000000Z-NA',
      collection_id='EO:EUM:DAT:MSG:CLM-IODC')
  
  try:
      print(selected_product) 
  except eumdac.product.ProductError as error:
      print(f"Error related to the product: '{error.msg}'")
  except requests.exceptions.ConnectionError as error:
      print(f"Error related to the connection: '{error.msg}'")
  except requests.exceptions.RequestException as error:
      print(f"Unexpected error: {error}")
  
  try:
      with selected_product.open() as fsrc, \
              open(fsrc.name, mode='wb') as fdst:
          shutil.copyfileobj(fsrc, fdst)
      print(f'Download of product {selected_product} finished.')
  except eumdac.product.ProductError as error:
      print(f"Error related to the product '{selected_product}' while trying to download it: '{error.msg}'")
  except requests.exceptions.ConnectionError as error:
      print(f"Error related to the connection: '{error.msg}'")
  except requests.exceptions.RequestException as error:
      print(f"Unexpected error: {error}")

• 从单个zip文件中下载需要的文件，如manifest.xml

  ## Download a single file from product
  for entry in selected_product.entries:
      try:
          print(entry)
      except eumdac.product.ProductError as error:
          print(f"Error related to the product: '{error.msg}'")
      except requests.exceptions.ConnectionError as error:
          print(f"Error related to the connection: '{error.msg}'")
      except requests.exceptions.RequestException as error:
          print(f"Unexpected error: {error}")
  
  ### Files within the product may be downloaded by providing the file's name, as demonstrated in the cell below.
  try:
      with selected_product.open(entry='manifest.xml') as fsrc, \
              open(fsrc.name, mode='wb') as fdst:
          shutil.copyfileobj(fsrc, fdst)
          print(f'Download of file {fsrc.name} finished.')
  except eumdac.product.ProductError as error:
      print(f"Error related to the product '{selected_product}' while trying to download it: '{error.msg}'")
  except requests.exceptions.ConnectionError as error:
      print(f"Error related to the connection: '{error.msg}'")
  except requests.exceptions.RequestException as error:
      print(f"Unexpected error: {error}")	

python 脚本

这里提供了一种可能的python脚本的示例。该脚本接受三个参数，八位日期，四位时间和存放路径，下载对应的grib2文件到指定路径。
Example to call in Fortan90

yyyymmdd="20221104"
HHMM="1445"
MSG_CLM_DIR='/data/jihenghu/data/MSG_CLM_IODC/'
CALL SYSTEM("python3.9 sub_download_msg_clm.py "//yyyymmdd//" "//HHMM//" "//trim(MSG_CLM_DIR))			

sub_download_msg_clm.py脚本内容如下：

import eumdac     ## pip3 istall eumdac
import datetime
import shutil
import requests
import os,sys

if len(sys.argv) < 3:
    print("ERROR! Non-adequate arguments for MSG Cloud Download Script !")
    exit()

yyyymmdd= sys.argv[1]
HHMM= sys.argv[2]
dirc= sys.argv[3]

## Download a single product

year=int(yyyymmdd[0:4])
month=int(yyyymmdd[4:6])
day=int(yyyymmdd[6:8])
hh=int(HHMM[0:2])
mm=int(HHMM[2:4])

dir_msg=dirc+'/'+yyyymmdd+'/'

if not os.path.exists(dir_msg):
    os.mkdir(dir_msg)


# Insert your personal key and secret into the single quotes
consumer_key = '#########################Qa'
consumer_secret = '#####################a'

credentials = (consumer_key, consumer_secret)

token = eumdac.AccessToken(credentials)

try:
    pass# print(f"This token '{token}' expires {token.expiration}")
except requests.exceptions.HTTPError as error:
    print(f"Unexpected error: {error}")

datastore = eumdac.DataStore(token)

# get collection
try:    
    # https://data.eumetsat.int/product/EO:EUM:DAT:MSG:CLM-IODC 
    selected_collection = datastore.get_collection('EO:EUM:DAT:MSG:CLM-IODC') 
    print(f"{selected_collection} - {selected_collection.title}")
except eumdac.datastore.DataStoreError as error:
    print(f"Error related to the data store: '{error.msg}'")
except eumdac.collection.CollectionError as error:
    print(f"Error related to the collection: '{error.msg}'")
except requests.exceptions.ConnectionError as error:
    print(f"Error related to the connection: '{error.msg}'")
except requests.exceptions.RequestException as error:
    print(f"Unexpected error: {error}")

# Set sensing start and end time
start = datetime.datetime(year, month, day, hh, mm)
end = datetime.datetime(year, month, day, hh, mm)

# Retrieve datasets that match our filter
products = selected_collection.search(dtstart=start, dtend=end)

for product in products:
    try:
        with product.open(entry=str(product)+'.grb') as fsrc, \
                open(dir_msg+fsrc.name, mode='wb') as fdst:
            shutil.copyfileobj(fsrc, fdst)
            print(f'Download of file {fsrc.name} finished.')
    except eumdac.product.ProductError as error:
        print(f"Error related to the product '{product}' while trying to download it: '{error.msg}'")
    except requests.exceptions.ConnectionError as error:
        print(f"Error related to the connection: '{error.msg}'")
    except requests.exceptions.RequestException as error:
        print(f"Unexpected error: {error}")

参考

• https://eumetsatspace.atlassian.net/wiki/spaces/DSDS/pages/328531972/Jupyter+Notebook+Using+the+Download+API
• https://gitlab.eumetsat.int/eumetlab/data-services/eumetsat_python_for_data_services
• https://api.eumetsat.int/api-key/