Deserializing `JArray`'s to numerical multidimensional C# arrays

I'm receiving varying JSONs with nested json objects and json arrays to C# app, and parsing them using Newtonsoft.JSON. I'm sending this data to MATLAB app, and want to send it as plain as I can for the MATLAB to parse the .NET types to MATLAB correctly.

For that, I convert the JObject's to Dictionary<string, dynamic>. I use dynamic so that the MATLAB will get int, double, String instead of object always.

I also convert the JArray's to object[] (using similar ToObject() function as in that thread).

What I'm missing is a way to fully convert JArray's to double[], int[,]. etc, if they are actually of that type. This will enable the MATLAB to parse them to MATLAB's double, int32, etc correctly.

Is there a way of achieving this? Can it also be applied to multidimensional arrays?

Clarification: If I'm getting arrays of strings, I don't mind leaving it object[]. I mostly care for numeric multidimensional arrays.

EDIT: As MATLAB prefer jagged arrays over multidimensional, so do I :)

Existing Code:

I currently expose the following Struct-like type to MATLAB, as it's not possible to marshal MATLAB structs to C#:

[JsonConverter(typeof(StructConverter))]
public class Struct
{

    [JsonExtensionData]
    Dictionary<string, dynamic> dict;

    public Struct() => dict = new Dictionary<string, dynamic>();
    public Struct(Dictionary<string, dynamic> d) => dict = d;
    public string[] GetKeys() => dict.Keys.ToArray();
    public void Set(string key, dynamic value) => dict[key] = value;
    public void SetVector(string key, int[] value) => dict[key] = value;
    public void SetVector(string key, double[] value) => dict[key] = value;
    public dynamic Get(string key) => dict[key];

}

Where StructConverter is my current converter:

public class StructConverter : JsonConverter
{

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        return ReadValue(reader);
    }

    private dynamic ReadValue(JsonReader reader)
    {
        switch (reader.TokenType)
        {
            case JsonToken.StartObject:
                return ReadObject(reader);
            case JsonToken.StartArray:
                return ReadArray(reader);
            default:
                return reader.Value;
        }
    }

    private dynamic ReadObject(JsonReader reader)
    {
        MatNet.Struct s = new MatNet.Struct();
        while (reader.Read())
        {
            switch (reader.TokenType)
            {
                case JsonToken.PropertyName:
                    string propName = reader.Value.ToString();
                    if (!reader.Read()) throw new Exception("Unexpected end when reading ExpandoObject.");
                    s.Set(propName, ReadValue(reader));
                    break;
                case JsonToken.Comment:
                    break;
                case JsonToken.EndObject:
                    return s;
            }
        }
        throw new Exception("Unexpected end when reading ExpandoObject.");
    }

    private dynamic ReadArray(JsonReader reader)
    {
        List<dynamic> list = new List<dynamic>();
        while (reader.Read())
        {
            switch (reader.TokenType)
            {
                case JsonToken.Comment:
                    break;
                default:
                    list.Add(ReadValue(reader));
                    break;
                case JsonToken.EndArray:
                    return list.ToArray();
            }
        }
        throw new Exception("Unexpected end when reading ExpandoObject.");
    }

}

I'm able to serialize and deserialize the following, but the arrays are of object[] instead of int[], string[] and double[]:

    var d = new MatNet.Struct();
    d.Set("a", new int[] { 1, 2, 4, 8, 16 });
    d.Set("b", new string[] { "qwer", "tyui", "asdf", "ghjk" });
    d.Set("c", new double[,] { {1, 3}, {2, 4}, {5, 7}, {6, 8} });
    d.Set("d", new MatNet.Struct());
    var json = JsonConvert.SerializeObject(d);
    dynamic x = JsonConvert.DeserializeObject<MatNet.Struct>(json);