经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置:技术经验 » Java相关 » Java » 查看文章
NativeBuferring,一种零分配的数据类型[下篇]
来源:cnblogs  作者:Artech  时间:2023/8/2 9:18:41  对本文有异议

上文说到Unmanaged、BufferedBinary和BufferedString是NativeBuffering支持的三个基本数据类型,其实我们也可以说NativeBuffering只支持UnmanagedIReadOnlyBufferedObject<T>两种类型,BufferedString、NativeBuffering和通过Source Generator生成的BufferedMessage类型,以及下面介绍的几种集合和字典类型,都实现了IReadOnlyBufferedObject<T>接口。

一、IReadOnlyBufferedObject<T>
二、集合
三、字典
四、为什么不直接返回接口?

一、IReadOnlyBufferedObject<T>

顾名思义,IReadOnlyBufferedObject<T>表示一个针对缓冲字节序列创建的只读数据类型。如下面的代码片段所示,该接口只定义了一个名为Parse静态方法,意味着对于任何一个实现了该接口的类型,对应的实例都可以利用一个代表缓冲字节序列的NativeBuffer的对象进行创建。

  1. public interface IReadOnlyBufferedObject<T> where T: IReadOnlyBufferedObject<T>
  2. {
  3. static abstract T Parse(NativeBuffer buffer);
  4. }
  5.  
  6. public unsafe readonly struct NativeBuffer
  7. {
  8. public byte[] Bytes { get; }
  9. public void* Start { get; }
  10.  
  11. public NativeBuffer(byte[] bytes, void* start)
  12. {
  13. Bytes = bytes ?? throw new ArgumentNullException(nameof(bytes));
  14. Start = start;
  15. }
  16.  
  17. public NativeBuffer(byte[] bytes, int index = 0)
  18. {
  19. Bytes = bytes ?? throw new ArgumentNullException(nameof(bytes));
  20. Start = Unsafe.AsPointer(ref bytes[index]);
  21. }
  22. }

由于IReadOnlyBufferedObject<T>是NativeBuffering支持的基础类型,而生成的BufferedMessage类型也实现了这个接口。通过这种“无限嵌套”的形式,我们可以定义一个具有任意结构的数据类型。比如我们具有如下这个表示联系人的Contact类型,我们需要利用它作为“源类型”生成对应BufferedMessage类型。

  1. [BufferedMessageSource]
  2. public partial class Contact
  3. {
  4. public Contact(string id, string name, Address address)
  5. {
  6. Id = id;
  7. Name = name;
  8. ShipAddress = address;
  9. }
  10.  
  11. public string Id { get; }
  12. public string Name { get; }
  13. public Address ShipAddress { get; }
  14. }
  15.  
  16. [BufferedMessageSource]
  17. public partial class Address
  18. {
  19. public string Province { get; }
  20. public string City { get; }
  21. public string District { get; }
  22. public string Street { get; }
  23. public Address(string province, string city, string district, string street)
  24. {
  25. Province = province ?? throw new ArgumentNullException(nameof(province));
  26. City = city ?? throw new ArgumentNullException(nameof(city));
  27. District = district ?? throw new ArgumentNullException(nameof(district));
  28. Street = street ?? throw new ArgumentNullException(nameof(street));
  29. }
  30. }

Contact具有Id、Name和ShipAddress 三个数据成员,ShipAddress 对应的Address又是一个复合类型,具有四个表示省、市、区和介绍的字符串类型成员。现在我们为Contact和Address这两个类型生成对应的ContactBufferedMessage和AddressBufferedMessage。

  1. public unsafe readonly struct ContactBufferedMessage : IReadOnlyBufferedObject<ContactBufferedMessage>
  2. {
  3. public NativeBuffer Buffer { get; }
  4. public ContactBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
  5. public static ContactBufferedMessage Parse(NativeBuffer buffer) => new ContactBufferedMessage(buffer);
  6. public BufferedString Id => Buffer.ReadBufferedObjectField<BufferedString>(0);
  7. public BufferedString Name => Buffer.ReadBufferedObjectField<BufferedString>(1);
  8. public AddressBufferedMessage ShipAddress => Buffer.ReadBufferedObjectField<AddressBufferedMessage>(2);
  9. }
  10.  
  11. public unsafe readonly struct AddressBufferedMessage : IReadOnlyBufferedObject<AddressBufferedMessage>
  12. {
  13. public NativeBuffer Buffer { get; }
  14. public AddressBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
  15. public static AddressBufferedMessage Parse(NativeBuffer buffer) => new AddressBufferedMessage(buffer);
  16. public BufferedString Province => Buffer.ReadBufferedObjectField<BufferedString>(0);
  17. public BufferedString City => Buffer.ReadBufferedObjectField<BufferedString>(1);
  18. public BufferedString District => Buffer.ReadBufferedObjectField<BufferedString>(2);
  19. public BufferedString Street => Buffer.ReadBufferedObjectField<BufferedString>(3);
  20. }

如下的程序演示了如何将一个Contact对象转换成字节数组,然后利用这这段字节序列生成一个ContactBufferedMessage对象。给出的调试断言验证了Contact和ContactBufferedMessage对象承载了一样的数据,fixed关键字是为了将字节数组“固定住”。(源代码从这里下载)

  1. using NativeBuffering;
  2. using System.Diagnostics;
  3.  
  4. var address = new Address("Jiangsu", "Suzhou", "Industory Park", "#328, Xinghu St");
  5. var contact = new Contact("123456789", "John Doe", address);
  6. var size = contact.CalculateSize();
  7. var bytes = new byte[size];
  8. var context = new BufferedObjectWriteContext(bytes);
  9. contact.Write(context);
  10.  
  11. unsafe
  12. {
  13. fixed (byte* _ = bytes)
  14. {
  15. var contactMessage = ContactBufferedMessage.Parse(new NativeBuffer(bytes));
  16. Debug.Assert(contactMessage.Id == "123456789");
  17. Debug.Assert(contactMessage.Name == "John Doe");
  18. Debug.Assert(contactMessage.ShipAddress.Province == "Jiangsu");
  19. Debug.Assert(contactMessage.ShipAddress.City == "Suzhou");
  20. Debug.Assert(contactMessage.ShipAddress.District == "Industory Park");
  21. Debug.Assert(contactMessage.ShipAddress.Street == "#328, Xinghu St");
  22. }
  23. }

二、集合

NativeBuffering同样支持集合。由于UnmanagedIReadOnlyBufferedObject<T>是两种基本的数据类型,它们的根据区别在于:前者的长度有类型本身决定,是固定长度类型,后者则是可变长度类型。元素类型为UnmanagedIReadOnlyBufferedObject<T>的集合分别通过ReadOnlyFixedLengthTypedList<T>和ReadOnlyVaraibleLengthTypedList<T>类型(结构体)表示,它们同样实现了IReadOnlyBufferedObject<T>接口。ReadOnlyFixedLengthTypedList<T>采用如下的字节布局:集合元素数量(4字节整数)+所有元素的字节内容(下图-上)。对于ReadOnlyVaraibleLengthTypedList<T>类型,我们会在前面为每个元素添加一个索引(4字节的整数),该索引指向目标元素在整个缓冲区的偏移量(下图-下)。

image

以如下所示的Entity为例,它具有两个数组类型的属性成员Collection1和Collection2,数组元素类型分别为Foobar和double,它们分别代表了上述的两种集合类型。

  1. [BufferedMessageSource]
  2. public partial class Entity
  3. {
  4. public Foobar[] Collection1 { get; }
  5. public double[] Collection2 { get; }
  6. public Entity(Foobar[] collection1, double[] collection2)
  7. {
  8. Collection1 = collection1;
  9. Collection2 = collection2;
  10. }
  11. }
  12.  
  13. [BufferedMessageSource]
  14. public partial class Foobar
  15. {
  16. public int Foo { get; }
  17. public string Bar { get; }
  18. public Foobar(int foo, string bar)
  19. {
  20. Foo = foo;
  21. Bar = bar;
  22. }
  23. }

NativeBuffering.Generator会将作为“源类型”的Entity和Foobar类型的生成对应的BufferedMessage类型(EntityBufferredMessage和FoobarBufferedMessage)。从EntityBufferredMessage类型的定义可以看出,两个集合属性的分别是ReadOnlyVariableLengthTypeList<FoobarBufferedMessage>和ReadOnlyFixedLengthTypedList<double>。

  1. public unsafe readonly struct EntityBufferedMessage : IReadOnlyBufferedObject<EntityBufferedMessage>
  2. {
  3. public NativeBuffer Buffer { get; }
  4. public EntityBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
  5. public static EntityBufferedMessage Parse(NativeBuffer buffer) => new EntityBufferedMessage(buffer);
  6. public ReadOnlyVariableLengthTypeList<FoobarBufferedMessage> Collection1 => Buffer.ReadBufferedObjectCollectionField<FoobarBufferedMessage>(0);
  7. public ReadOnlyFixedLengthTypedList<System.Double> Collection2 => Buffer.ReadUnmanagedCollectionField<System.Double>(1);
  8. }
  9.  
  10. public unsafe readonly struct FoobarBufferedMessage : IReadOnlyBufferedObject<FoobarBufferedMessage>
  11. {
  12. public NativeBuffer Buffer { get; }
  13. public FoobarBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
  14. public static FoobarBufferedMessage Parse(NativeBuffer buffer) => new FoobarBufferedMessage(buffer);
  15. public System.Int32 Foo => Buffer.ReadUnmanagedField<System.Int32>(0);
  16. public BufferedString Bar => Buffer.ReadBufferedObjectField<BufferedString>(1);
  17. }

两个集合类型都实现了IEnumerable<T>接口,还提供了索引。下面的代码演示了以索引的形式提取集合元素(源代码从这里下载)。

  1. using NativeBuffering;
  2. using System.Diagnostics;
  3.  
  4. var entity = new Entity(
  5. collection1: new Foobar[] { new Foobar(1, "foo"), new Foobar(2, "bar") },
  6. collection2: new double[] { 1.1, 2.2 });
  7. var bytes = new byte[entity.CalculateSize()];
  8. var context = new BufferedObjectWriteContext(bytes);
  9. entity.Write(context);
  10.  
  11. unsafe
  12. {
  13. fixed (byte* p = bytes)
  14. {
  15. var entityMessage = EntityBufferedMessage.Parse(new NativeBuffer(bytes));
  16. var foobar = entityMessage.Collection1[0];
  17. Debug.Assert(foobar.Foo == 1);
  18. Debug.Assert(foobar.Bar == "foo");
  19.  
  20. foobar = entityMessage.Collection1[1];
  21. Debug.Assert(foobar.Foo == 2);
  22. Debug.Assert(foobar.Bar == "bar");
  23.  
  24. Debug.Assert(entityMessage.Collection2[0] == 1.1);
  25. Debug.Assert(entityMessage.Collection2[1] == 2.2);
  26. }
  27. }

三、字典

从数据的存储来看,字典就是键值对的集合,所以我们采用与集合一致的存储形式。NativeBuffering对集合的Key作了限制,要求其类型只能是Unmanaged字符串(String/BufferredString)。按照Key和Value的类型组合,我们一共定义了四种类型的字典类型,它们分别是:

  • ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue>:Key=Unmanaged; Value = Unmanaged
  • ReadOnlyUnmanagedBufferedObjectDictionary<TKey, TValue>:Key=Unmanaged; Value = IReadOnlyBufferedObject<TValue>
  • ReadOnlyStringUnmanagedDictionary<TValue>:Key=String/BufferredString; Value = Unmanaged
  • ReadOnlyStringBufferedObjectDictionary<TValue>:Key=String/BufferredString; Value = IReadOnlyBufferedObject<TValue>

如果Key和Value的类型都是Unmanaged,键值对就是定长类型,所以我们会采用类似于ReadOnlyFixedLengthTypedList<T>的字节布局方式(下图-上),至于其他三种字典类型,则采用类似于ReadOnlyVaraibleLengthTypedList<T>的字节布局形式(下图-下)。

image

但是这仅仅解决了字段数据存储的问题,字典基于哈希检索定位的功能是没有办法实现的。这里我们不得不作出妥协,四种字典的索引均不能提供时间复杂度O(1)的哈希检索方式。为了在现有的数据结构上使针对Key的查找尽可能高效,在生成字节内容之前,我们会按照Key对键值对进行排序,这样我们至少可以采用二分法的形式进行检索,所以四种类型的字典的索引在根据指定的Key查找对应Value,对应的时间复杂度为Log(N)。如果字典包含的元素比较多,这样的查找方式不能满足我们的需求,我们可以I将它们转换成普通的Dictionary<TKey, TValue>类型,但是这就没法避免内存分配了。

我们照例编写一个简答的程序来演示针对字典的使用。我们定义了如下这个Entity作为“源类型”,它的四个属性对应的字典类型刚好对应上述四种键值对的组合。从生成的EntityBufferedMessage类型可以看出,四个成员的类型正好对应上述的四种字典类型。

  1. [BufferedMessageSource]
  2. public partial class Entity
  3. {
  4. public Dictionary<int, long> Dictionary1 { get; set; }
  5. public Dictionary<int, string> Dictionary2 { get; set; }
  6. public Dictionary<string, long> Dictionary3 { get; set; }
  7. public Dictionary<string, string> Dictionary4 { get; set; }
  8. }
  9.  
  10. public unsafe readonly struct EntityBufferedMessage : IReadOnlyBufferedObject<EntityBufferedMessage>
  11. {
  12. public NativeBuffer Buffer { get; }
  13. public EntityBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
  14. public static EntityBufferedMessage Parse(NativeBuffer buffer) => new EntityBufferedMessage(buffer);
  15. public ReadOnlyUnmanagedUnmanagedDictionary<System.Int32, System.Int64> Dictionary1 => Buffer.ReadUnmanagedUnmanagedDictionaryField<System.Int32, System.Int64>(0);
  16. public ReadOnlyUnmanagedBufferedObjectDictionary<System.Int32, BufferedString> Dictionary2 => Buffer.ReadUnmanagedBufferedObjectDictionaryField<System.Int32, BufferedString>(1);
  17. public ReadOnlyStringUnmanagedDictionary<System.Int64> Dictionary3 => Buffer.ReadStringUnmanagedDictionaryField<System.Int64>(2);
  18. public ReadOnlyStringBufferedObjectDictionary<BufferedString> Dictionary4 => Buffer.ReadStringBufferedObjectDictionaryField<BufferedString>(3);
  19. }

如下的代码演示了基于四种字典类型基于“索引”的检索方式(源代码从这里下载)。

  1. using NativeBuffering;
  2. using System.Diagnostics;
  3.  
  4. var entity = new Entity
  5. {
  6. Dictionary1 = new Dictionary<int, long> { { 1, 1 }, { 2, 2 }, { 3, 3 } },
  7. Dictionary2 = new Dictionary<int, string> { { 1, "foo" }, { 2, "bar" }, { 3, "baz" } },
  8. Dictionary3 = new Dictionary<string, long> { { "foo", 1 }, { "bar", 2 }, { "baz", 3 } },
  9. Dictionary4 = new Dictionary<string, string> { { "a", "foo" }, { "b", "bar" }, { "c", "baz" } }
  10. };
  11.  
  12. var bytes = new byte[entity.CalculateSize()];
  13. var context = new BufferedObjectWriteContext(bytes);
  14. entity.Write(context);
  15. unsafe
  16. {
  17. fixed (void* _ = bytes)
  18. {
  19. var bufferedMessage = EntityBufferedMessage.Parse(new NativeBuffer(bytes));
  20.  
  21. ref var value1 = ref bufferedMessage.Dictionary1.AsRef(1);
  22. Debug.Assert(value1 == 1);
  23. ref var value2 = ref bufferedMessage.Dictionary3.AsRef("baz");
  24. Debug.Assert(value2 == 3);
  25.  
  26. var dictionary1 = bufferedMessage.Dictionary1;
  27. Debug.Assert(dictionary1[1] == 1);
  28. Debug.Assert(dictionary1[2] == 2);
  29. Debug.Assert(dictionary1[3] == 3);
  30.  
  31. var dictionary2 = bufferedMessage.Dictionary2;
  32. Debug.Assert(dictionary2[1] == "foo");
  33. Debug.Assert(dictionary2[2] == "bar");
  34. Debug.Assert(dictionary2[3] == "baz");
  35.  
  36. var dictionary3 = bufferedMessage.Dictionary3;
  37. Debug.Assert(dictionary3["foo"] == 1);
  38. Debug.Assert(dictionary3["bar"] == 2);
  39. Debug.Assert(dictionary3["baz"] == 3);
  40.  
  41. var dictionary4 = bufferedMessage.Dictionary4;
  42. Debug.Assert(dictionary4["a"] == "foo");
  43. Debug.Assert(dictionary4["b"] == "bar");
  44. Debug.Assert(dictionary4["c"] == "baz");
  45. }
  46. }

四、为什么不直接返回接口

针对集合,NativeBuffering提供了两种类型;针对字典,更是定义了四种类型,为什么不直接返回IList<T>/IDictionary<TKey,TValue>(或者IReadOnlyList<T>/IReadOnlyDictionary<TKey,TValue>)接口呢?这主要有两个原因,第一:为了尽可能地减少内存占用,我们将四种字典类型都定义成了结构体,如果使用接口的话会导致装箱;第二,四种字典类型的提供的API是有差异的,比如ReadOnlyFixedLengthTypedList<T> 和ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue>都提供了一个额外的AsRef方法,它直接返回值的引用(只读)。如果这个值被定义成一个成员较多的结构体,传引用的方式可以避免较多的拷贝。

  1. public readonly unsafe struct ReadOnlyFixedLengthTypedList<T> : IReadOnlyList<T>, IReadOnlyBufferedObject<ReadOnlyFixedLengthTypedList<T>>
  2. where T: unmanaged
  3. {
  4. public readonly ref T AsRef(int index);
  5. ...
  6. }
  7.  
  8. public unsafe readonly struct ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue> : IReadOnlyDictionary<TKey, TValue>, IReadOnlyBufferedObject<ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue>>
  9. where TKey : unmanaged, IComparable<TKey>
  10. where TValue : unmanaged
  11. {
  12. public readonly ref TValue AsRef(TKey index) ;
  13. ...
  14. }

原文链接:https://www.cnblogs.com/artech/p/17587660.html

 友情链接:直通硅谷  点职佳  北美留学生论坛

本站QQ群:前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

W3xue 的所有内容仅供测试,对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接(目前9元/月)请联系QQ:27243702 沸活量
皖ICP备17017327号-2 皖公网安备34020702000426号